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Abstract: This special issue collects contributions from the participants of the “Information 
in Dynamical Systems and Complex Systems” workshop, which cover a wide range of 
important problems and new approaches that lie in the intersection of information theory and 
dynamical systems. The contributions include theoretical characterization and understanding 
of the different types of information flow and causality in general stochastic processes, 
inference and identification of coupling structure and parameters of system dynamics, 
rigorous coarse-grain modeling of network dynamical systems, and exact statistical testing of 
fundamental information-theoretic quantities such as the mutual information. The collective 
efforts reported herein reflect a modem perspective of the intimate connection between 
dynamical systems and information flow, leading to the promise of better understanding 
and modeling of natural complex systems and better/optimal design of engineering systems. 
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1. Introduction 

Erom 18-19 July 2013, a workshop was held, entitled, “Information in Dynamical Systems and 
Complex Systems Summer 2013 Workshop” in Burlington, VT, with Organizers: Erik M. Bollt and 
Jie Sun (Clarkson University). This special issue of Entropy represents collective thoughts on the 
workshop themes stated as follows. Given the modern focus of dynamical systems on coupled oscillators 
that form complex networks, it is important to move forward and explore these problems from the 
perspective of information content, flow and causality. The following general themes were discussed: 
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1. Information flow. In particular transfer entropy has gained a great deal of interest in reeent years 
as future states may be eonditioned on past states both with and without aeeess to other stoehastie 
proeesses as a test through Kullbaek-Leibler divergenee, but reeent work suggests that there are 
possible misinterpretations from the use of transfer entropy for eausality inferenee. 

2. Causality, and information signatures of eausation. A eentral question in seienee is what eauses 
outeomes of interest, and in partieular for affeeting and eontrolling outeomes this is even more 
important. From the perspeetive of information flow, eausation and eausal inferenee beeomes 
partieularly poignant. 

3. Symmetries and reversibility, whieh may be exploited in speeial eireumstanees to enlighten 
understanding of oausality,struoture, as well as elustering. 

4. Seales, hierarehies, lead to understanding of relevanee and nested topologieal partitions when 
defining a seale from whieh information symbols are determined. Variational and optimization 
prineiples of information in physies, in partieular regarding maximum entropy prineiples that lead 
to the understanding of underlying laws. 

5. Randomness, strueture and eausality. In some sense randomness may be deseribed as external and 
unmodelled effeets, whieh we may interpret in the eontext here as unknown information. This leads 
to: hidden states and hidden proeesses, ineluding sueh methods as hidden Markov models and 
more generally Bayesian inferenee methods. In the eontext of information eontent of a dynamieal 
system, sueh perspeetive should potentially yield better understanding. 

6. Measures and metries of eomplexity and information eontent. The phrase ""complexity'" is 
eommonly used for a wide variety of systems, behaviors and proeesses, and yet a eommonly 
agreed deseription as to what the phrase means is laeking. 

7. Physieal laws as information filters or algorithms. Sinee physieal laws lead to evolution equations, 
whieh from the perspeetive of this diseussion defines evolution from some information state to a 
new information state, then it ean be said that physieal laws may be deseribed either as algorithms 
or information filters that translate states. 

To this end the papers in this speeial issue we now summarize and as we see they point elearly at these 
key problems. 

In the paper by Sarah Marzen and James R Crutehfield, “Information Anatomy of Stoehastie 
Equilibria” (in this issue, Referenee [1]) there is a beautiful paper deseribing many different measures 
following entropy rate of information eontent in eoupled stoehastie proeesses. Speeifieally, eonsidering 
entropy as a measure of randomness, or said otherwise predietability, the basie question is what must be 
known about past states of a stoehastie proeess in order to make the future predietable. Furthermore they 
diseuss the role of memory as stored in easual states. Their program of deseription they eall “information 
anatomy” wherein several quantities are ineluded: bound information deseribes information between 
present and past as a mutual information. Elusive information quantifies predietable information 
not eaptured in the present, in other words how mueh the past eouples to the future, again as an 
appropriately eonditioned mutual information. This relates to possible hidden states and the speeifie 
mode of observation. Further, ephemeral information links uneertainty in the present to past and 
present, and bound information links information shared in present and future eonditioned on past. 
Finally enigmatie information is a joint information of past, present and future. With these quantities so 
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defined, then there are several immediate equations that link these quantities, and this is so-ealled the 
information anatomy. While these were previously defined by the authors [2], here it was extended to 
finite-order non-autonomous Markov proeesses, and also eontinuous-valued proeesses. Many beautiful 
and insightful perspeetives eome in our broader understanding of stoehastie proeesses with these many 
information theoretie eonneetions between past, present and future in a stoehastie proeess. For example, 
while the general information anatomy may allow predictability, there are instances described where the 
present information can actually shield past information from the future. Other useful results include 
such as a Markov assumption implies zero elusive entropy, and many other such descriptive results 
about casual inference. A good deal of rigorous examples are produced for the Langevin processes. An 
exciting observation for future work is stated that an information anatomy perspective leads to a trade-off 
between information storage and a heat-loss description suggestive of a Maxwell’s demon. 

In “Coarse Dynamics for Coarse Modeling: An Example From Population Biology” by Bush and 
Mischaikow (in this issue. Reference [3]) the goal is to rigorously define a coarse grained graph model 
of a dynamical system. Furthermore the model should be robust in the sense that it should remain valid 
across a subset of parameter variations. In this sense the model should be structurally stable in some 
sense, but to do this, a qualitative model is produced and perhaps valid on a subset of the phase space, and 
furthermore as the title states, it represents a coarser picture of the dynamics. By coarse the authors mean 
that on larger scale of the measurements in the phase space, it is valid that there are orbits that behave 
as the coarse grain dynamics predicts. In this sense, the modeling reminds of a symbolic dynamical 
system, and the information theoretic description is implicit, although there are key differences. While 
the modeling is admittedly crude, its strength is that it is robust across a parameter range. The methods 
of developing a database to rigorously describe the global dynamics across a range of parameters is to 
develop a Morse decomposition leading to a so-called Hasse diagram [4,5]. The goal is to identify the 
recurrent behavior by identifying strongly connected path components. Eventually this should lead to 
developing an algebraic topological invariant, namely the Conley index. For now, these methods can be 
taken to be an approach of bookkeeping the “database” of the robust part of the coarse dynamics. To 
demonstrate the computational efficacy of their methods, the authors illustrate with a model of population 
dynamics on gene regulatory/signal transduction networks. Further several other interesting dynamical 
scenarios are demonstrated such as bistability in a population dynamics of fish species. Perhaps a 
take-away message here could be that a database approach to dynamical systems while coarse makes 
for a high level summary that could be most useful for certain policy scenarios. 

The paper, herein [6], by Pethel and Hahs, “Exact significance test for mutual information,” develops 
a key step that is crucial for any serious data-driven estimator of several of the entropies discussed in 
this issue and well beyond. That is the authors develop the first exact null hypothesis significance test 
for mutual information under the presence of temporal dependencies (Markov structure). Their null 
hypothesis is that the mutual information is in fact zero. This is critical in practice since typically when 
developing an estimator of mutual information, whether most crudely by binning methods [7] or by 
more sophisticated kernel methods [8-11], often one deals with a great deal of data which computes 
too many estimated values of mutual information many of which are small. One must decide if they are 
simply small but nonzero, or the values are insignificant and therefore should be considered (truncated) to 
zero. Their approach uses a novel resampling technique leading to surrogates that preserve the Markov 
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structure of the original data (whereas most heuristic methods do not and often lead to fundamental 
errors). From the surrogate data, the null hypothesis is then extracted. The method is shown to be 
applicable for general finite-order Markov processes where the order must be first determined from 
another test (exact significance test for Markov order) developed by the same authors in Reference [12]. 
This was illustrated through data generated from the logistic maps, by considering optimal order using 
block entropies compared to that of the next highest order. This work is surely an important step toward 
the outstanding goal of developing an exact significance test for conditional mutual information, since it 
is this quantity on which transfer entropy, causation entropy, and the several varieties of entropy part of 
the information anatomy are based, as found in several of the papers in this issue. 

The next paper in this special issue is “Infinite Excess Entropy Process with Countable-State 
Generators” by Travers and Crutchfield (in this issue. Reference [13]). Therein recalled the definition of 
the excess entropy between the complete infinite past and complete infinite future of a stochastic process, 
this quantity has been used as a classifier as “finitary” when finite and otherwise as “infinitary” and this is 
used as a measure of correlation and complexity. While a popular measure, most studies have focused on 
finite alphabet systems, most of which have shown to be finitary such as finite-order Markov processes 
and processes with finite-state hidden Markov models. It is known that symbolic dynamics at the onset of 
chaos can be infinitary systems. Here for the first time, building on a result from “the Santa Ee process”, 
examples of countable-state hidden Markov models are constructed that generate finite-alphabet but are 
in fact infinitary processes. In so doing not only do they offer first time new examples of this scenario, 
but also further extend the notions of predictability and therefore entropy connecting past to future. 

The paper herein, “Simultaneous State and Parameter Estimation using Maximum Relative Entropy 
with Nonhomogenous Differential Equation Constrains," by Giffin and Umiezius, [14] takes a new 
approach to the classic problem of filtering from a dynamical system. Specifically the problem is given 
an observed (possibly multi-variate) time-series from a differential equation, and with possibly noise on 
the observations, but assuming a general form for the differential equation that produced the signal, but 
with unknown parameter values in the system, can the a “de-noised" signal and best estimates of the 
parameters be inferred? Generally, the problem of on-line estimation is known as filtering, and most 
popularly the Kalman filters have had great adoption, and to the problem at hand the extended Kalman 
filter, perhaps with a pre-filtering would come to mind. But as pointed out here, there are underlying 
assumptions of memorylessness (Markov) the must be met for the Kalman approach, not to mention 
difficulties for nonlinear systems and systems with far from equilibrium. If the question is to estimate 
states and parameters simultaneously and without a Markov assumption, particularly for a nonlinear 
system, here a new approach is suggested based on the “maximum relative entropy" (MrE) formalism. 
This allows an assumed model form to together with prior assumptions on state and parameter estimates 
to be incorporated and then updated on the fly based on observables, and not requiring the Markov 
assumptions of the traditional methods. Since the MrE is essentially a constrained optimization method, 
a Eagrange multiplier approach is standard and shown here to lead to useful and quickly implemented 
update schemes. An example with a (low-dimensional) RE circuit with some essential information 
regarding parameter and model form incorporated into the scheme are shown here to lead to excellent 
estimates. Thus here entropic methods lead to a practical and important new direction in an important 
and widely used method in applied engineering, namely filtering. 
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Interestingly, as reported in the paper “Identifying Chaotie FitzHugh-Nagumo Neurons Using 
Compressive Sensing” by by Su, Lai, and Wang (herein, Referenee [15]), another perspeetive of 
ineorporating prior knowledge/information finds applieation in the identifieation of system dynamies. In 
partieular, by assuming eomplete observability and knowledge of the terms/funetions that ean potentially 
arise in the equations under a noise-free setting, the problem of nonlinear identifieation is eonverted into 
a linear one with respeet to the basis funetions. Then, the resulting linear inverse problem is solved by the 
well-developed “£i magie”, i.e., minimizing error in terms of the ii norm rather than £ 2 . This ii approaeh 
was initially developed for the reeovery of sparse signals when the measurement matrix satisfies the 
uniform uncertainty principle (sometimes also referred to as the restricted isometry hypothesis) [16-18]. 
Curiously, although it remains unelear whether or not sueh prineiple/hypothesis was satisfied for the 
effeetive measurement matrix that arises after the nonlinear-to-linear transformation, Referenee [15] 
still finds the £i solution to be quite satisfaetory, as illustrated for the synthetie network of nonlinear 
FitzHugh-Nagumo (neuron) oseillators with linear eoupling [19,20]. This eneouraging finding raises an 
interesting question about whether one eould in faet extend the established “uneertainty prineiples” to 
the nonlinear, potentially basis-free seenario. 

Clearly as revealed by the information anatomy diseussions, the eonneetions between past and 
future is a most fundamental question in any stoehastie proeess as viewed from an information theory 
perspeetive. Sinee the development of transfer entropy by Sehreiber [21] (also see Referenee [22]) a 
great deal of attention has been dealt to questions of how information flows from a stoehastie system 
with (at least) two elements. In the paper by Butail, Ladu, Spinello, and Porfiri, “Information Flow in 
Animal-Robot Interaetions” (in this issue, Referenee [23]), the questions of how groups of animals or 
groups of animals and a robot ean eoordinate their aetions is studied with the tools of information theory. 
Namely it is asked what information may flow between these agents, as measured by transfer entropy. 
This is both an experimental and theoretieal study. An experiment with zebra fish is designed together 
with a fish analogue robot and the trajeetories of eaeh are eompared and the possible interaetions are 
measured by transfer entropies. If transfer entropies are a Kullbaek-Leibler divergenee between fish 
evolution eonditioned on its own past, and eonditioned on its own past and that of the external agent - 
the robot, then this is meant to deseribe if information from the robot influenees the fish. In faet, there is 
an information flow found, as measured in bits per time units, and the flow is found to be bi-direetional 
and the signifieanee of the measurements is found to be nontrivial. This represents a unique and new 
direetion to the broader questions of soeial interaetions but speoifieally here it may bring to bear a strong 
method to deeide the “simple” and fundamental, but hard to answer question of “who follows whom”. 

In the paper “Identifying Coupling Strueture in Complex Systems through the Optimal Causation 
Entropy Prineiple” by Sun, Cafaro, and Bollt herein [24], the authors diseussed the general problem of 
eausality inferenee in eomplex systems from time series data based on information-theoretie tools. In 
partieular, the authors reviewed a new theoretieal quantity ealled Causation Entropy introdueed in [25] 
and subsequent developments in [26]. Causation Entropy ean be used to measure direet and indireet 
information flow among the eomponents within a eomplex system by various appropriate eonditioning. 
This was shown rigorously and explieitly in Referenee [26], summarized as the Optimal Causation 
Entropy Prineiple whieh states that the direet eausal influenees of a given eomponent in a system is 
the unique minimal set of eomponents that maximizes the (uneonditional) Causation Entropy to that 
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component. This allows the development of effieient algorithms to infer cause-and-effect relationships 
without running into the eommon issue of having false positives (at least in prineiple). In the eontributed 
work, the authors present an applieation of the Optimal Causation Entropy Prineiple to infer the direet 
eouplings of the repressilator [27], whieh is a synthetic biological system modeling the dynamies of the 
eoneentration of three distinet mRNAs and their eorresponding proteins. The time series data is obtained 
by applying stochastic perturbations to the system at equilibrium and recording both the perturbations 
and the system’s responses shortly after each such perturbation. The authors show that by using the 
algorithms developed based on the Optimal Causation Entropy Prineiple, the aecuracy of eoupling 
inferenees beeomes higher when more data are available and also increases with higher frequeney of 
sampling and is espeeially immune to false positives. 

Einally, the paper “Cross-scale interaetions and information transfer” (herein, Referenee [28]) by 
Palus focuses on a simply stated but often overlooked and extremely interesting problem: given times 
series data of a single system that evolves on multiple time seales, how to measure the direet information 
flow and deteet eausal relationships between the dynamics on these different time seales? The reeipe 
proposed by Palus first deeomposes the measured signal into a number of modes that eorrespond to 
some speeifie amplitudes and phases (for example, via a wavelet transform). Then, the information 
flow and therefore potential direeted eausalities are inferred via the estimation of (eonditional) mutual 
information between the respective phase/amplitude signals. The proposed method was benehmarked 
via a model multifraetal proeess whieh exhibits information transfer from larger to smaller time seales, 
and further, applied to a long time series of surfaee air temperature from various European sites. In 
the latter ease no eross-seale phase-phase interactions were found, while slow elimate oseillations were 
deteeted to (direetly or indireetly) influenee the future dynamies of the fast temperature variability from 
an information-theoretie perspective. It is our sense that the many insightful ideas and discussions 
presented in Reference [28] are likely to motivate future development of a coherenee framework to define 
and identify information flow and eausality in general spatial-temporal multi-scale stoehastie dynamies. 

At the outset of the workshop and as the eall to this speeial issue, several questions were asked, some 
of which we repeat here, now with a post-script interpretation of hindsight that are eumulatively pulled 
from both the papers submitted here and also the discussions from Burlington: 

1. Can we develop a general mechanistie deseription of what renders a real complex system different 
from a large but perhaps simpler system (particularly from an information theory perspeetive)? 

• This question eould eventually lead to many direetions aeross many seientific fields, and 
the group agreed that this was a problem worth emphasizing. Indeed, the concept is weaved 
deeply into several of the papers here. If there is a unifying theme, then the paper [3] suggests 
a skeleton or eoarse structure aeross parameters. On the other hand the casual structure [24] 
suggests eonneetions between many elements, but possibly when sueh structure is 
hierarehieal then this should be eonsidered as eomplex meaning multi-sealed as opposed 
to simply a large homogenous system. Einally, the question of Unitary versus infinitary 
examples in [13] suggests an enumerable differenee between different types of complex 
systems, not to mention the information anatomy [1] allows a detailed description of how 
past, present, and future influenee eaeh other, and perhaps when eonsidered also eonditioned 
aeross spatial seales, then this eould be the basis of sueh an interpretation. 
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2. Can physical laws be defined in an algorithmic and information theoretic manner? 

• Clearly this will be a deeply important connection between physics, physical laws that 
describe governance of measurable quantities, and the knowledge of states that are governed 
by these laws. Since knowledge of states represents information, the specific physical laws 
that evolve these states is a flow of information, and the information anatomy [1] perhaps 
gets the bottom of the story of how past, present and future are coupled. Can the physics be 
inferred directly from the information theory? This would be a new paradigm to discovery 
to which we have not yet attained but the question remains extremely important. Partial 
steps are made in the sense that causal inferences are discussed in [24] and also a parametric 
approach for assumed model forms is the classic filtering problem as discussed in [14]. 

3. Identify engineering applications, especially those that benefit directly from information theory 
perspective and methods. How can this perspective impact design? Can specific control methods 
be developed that benefit? 

• This is the engineering repeat of the previous question, if engineering means designing a 
problem with desired properties. The connection between physics and information theory 
and an algorithmic formalism between them would suggest that if there is an desired 
engineering design then a good understanding of the information theory should be beneficial. 
Simply from an observational stand point, the study of fish (agents) and robot [23] has the 
engineering perspective in that the experimental design allows for studies of how adjusting 
the robots behavior results in the change of the agents behaviors, and how this influences and 
influenced by the information flow between them. This then hints at a possible engineering 
approach for this general question in the future. 

4. Can group behaviors and cooperative behaviors such as those of animals and humans be better 
understood in terms of information theoretic descriptions? What role does hierarchical structures 
come into play? 

• Clearly the paper by [23] tackles this question directly by demonstrating that group behavior 
and cooperation as having an underlying information flow that can be measured and adjusted. 
What would remain now is to consider hierarchical aspects of group behaviors, perhaps in 
coarse grains. 

5. Can methods designed to identify causal influences be adapted to further adjust and define control 
strategies for complex systems in biological, social, physical and engineering contexts? 

• As the previous question, this asks how can the information theory perspective be turned 
around for engineering questions of design. The casual inference studied in [24] gives the 
formal definitions and methods to define what it means to have control strategies that may 
causally influence states, but does not yet describe the inverse problems necessary for an 
engineering control strategy asked here. How can information flow be a design feature to 
affect desired and controlled states? This will be a question surely for future technological 
innovation. That said, and with the information flow studies of [23] and information anatomy 
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discussed in [13], all of the elements are here to at least well define the question. Design of 
a solution here will likely lead to new teehnologies not even previously dream of. 

6. Is there a minimal information deseription of a dynamieal system that will faeilitate engineering 
design? Does approximate deseription of the formal language suffiee for approximate modeling 
lead to faster and easier design? 

• This question lies underlying essentially all of the papers here in, and eentral to our 
diseussion at the workshop. The information in a deseription may be interpreted as imposing 
a partition of states on a stoehastie system, and onee the partition is imposed (see for example 
Referenee [29]) the measure on states may follow from whieh any information theoretie 
quantity may be eomputed. Therefore the question of minimal information deseription maps 
to a question of an infemum of the relevant funetional, over all possible partitions, and 
with respeet to all measures on eaeh partition. This is elearly eomputationally impossible 
to approaeh by brute foree but the eoneept seems likely to lie at the heart of understanding 
a proeess. It is interesting to note that in topologieal dynamies there exists a variational 
deseription of the topologieal entropy whieh ean be deseribed in terms of the “maximal 
entropy measure” [30,31] whieh in that setting eorresponds to the positive answer this 
question. On the other hand, this same question may be interpreted eonstruetively; how 
mueh information is needed to build or design a given dynamieal system. This brings the 
question eloser to that of Kolmogorov eomplexity [32,33] whieh speeialized to this question 
here, is how mueh information is needed to eneode the algorithm that produees the observed 
data. That question is in general unanswered. 

As a summary statement, it was understood that the validity of the popular approaehes of information 
and entropy measures as a system’s probe, deseription, and summary. A wide variety of entropie 
prineiples have beeome popular lately in a good deal of reeent literature, and therefore at this stage it 
would be most useful to pause and reeonsider whieh of these quantities imply the various relationships. 
A elear voiee and a summary of eonneetions between these eoneepts is erueial for this seientifie endeavor 
to proeeed properly and it is hoped that this speeial issue will help add to this elarity. 

Dr. Erik Bollt 
Dr. lie Sun 
Guest Editors. 
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